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Abstract 

Phelan (2000) has produced a complex bibliometric analysis of the international 
contribution of Australian educational research based upon publications and 
citations reported in the journals indexed by the Institute for Scientific Information - 
the Standard & Poors of the academic world. This paper examines Phelan ’s analysis, 
showing its strengths and weaknesses, as well as examining his proposal for the 
establishment of an Australian database along the lines of the ISI’s index. 

Bibliometric analysis is an imprecise art, as Phelan (2000) acknowledges. It is 
frequently dependent on databases that are constructed on principles that do not 
serve the bibliometrician’s purposes; which do not combine easily with other 
databases or which vary in time period or coverage. In his study of the productivity 
and impact of Australian educational research Phelan acknowledges these limitations 
but also suggests that there is sufficient strength in the data he uses to produce a 
reliable estimate of Australia’s international research reputation. 

His publication and citation data is drawn from the list of journals indexed by the 
Institute for Scientific Information, a US-based organisation that lists publications and 
paper citations for ‘leading journals’ in most scientific fields. From the overall database 
only those publications and citations with Australian authors from 1981-97 were 
selected through the Research and Evaluation Policy project. This produced 
information on 300 000 papers written by some 800 000 authors, published in some 
2693 journals. When classified by field the rather surprising conclusion was that of 
the thirteen major fields of publication 1 only earth sciences (which includes marine 
sciences and geology) had a higher percentage of world publications than education. 
In Phelan’s words, 

the contribution of education to the world literature is surprisingly 

strong compared to other fields of research. Among this set of major 
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fields, education is the second largest contributor in terms of publication. 

That share has also grown in recent years at a faster rate than it has for 
other fields. (Phelan 2000, p. 608) 

If that is the case in comparison to other fields of research in Australia it is also the case 
that Australia seems to be fighting above its weight in the international educational 
literature. Taking the 35 most prolific countries for comparison, Phelan concludes: 

Producing about 3.5 per cent of international publications in the area of 
education, Australia was the fourth largest producer in this field for the 
period from 1987 through June 1998. This is a very strong performance 
for a relatively small country. Only the United States (66.8 per cent), the 
United Kingdom (10.00 per cent) and Canada (5.6 per cent) produced 
more educational research than Australia and all three of these countries 
are considerably larger both in terms of population and in terms of the 
size of their economies. (Phelan 2000, p. 595) 

Notably, Australia, on these statistics, also outperformed all European countries as well 
as Japan. But before we take this assessment as the basis for self-congratulation it would 
be well to look a little more closely at the data on which these judgements are made. 

Phelan fortunately provides a table listing the journals included in the ISI list. Not all of 
them are easily identified from the title abbreviations but of the 169 listed it is possible 
to readily identify 143. Of these 94 (66 per cent) were published in the USA, 33 (23 per 
cent) in the United Kingdom, 8 (5 per cent) in Europe, 3 (2 per cent) in Russia, 2 (1 
per cent) in Canada and 1 (.6 per cent) in Australia, New Zealand and China 
respectively. Even if Europe is disaggregated into the three contributing countries it 
becomes apparent that journals from only nine countries are represented here despite 
the fact that judgements are being made about the educational productivity of some 
thirty five. 

Moreover the criteria of inclusion in the ISI list are unclear. This may partly be the result 
of difficulties in defining the field of education. A strict definition (journals related to 
primary, secondary and tertiary education across curricular, pedagogical and assessment 
fields as well as policy and management, for instance) would clearly exclude a 
considerable number of journals with educational interests. A very loose definition 
might however also include journals with marginal relationship to ‘mainstream’ 
education. The ISI list for instance includes journals on medical and legal education, 
academic psychiatry and academic medicine. A quantitative solution to this issue of 
relative importance is possible in terms of circulation numbers but this might exclude 
some very important and influential specialist journals as well as producing a significant 
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bias towards journals produced in countries with large populations. 2 It is also that case 
that educational research is often published in ‘non-education’ journals. 

It is also clear that educational researchers publish most of their research in local’ 
journals. For instance 75 per cent of Australian publications are in non-ISI-listed 
journals. Phelan is aware of this problem, particularly in terms of a bias towards English- 
language journals in the ISI index. He uses India as an example. 

Thus, for example, while India produces only just over one third of one 

per cent of articles appearing in these (ISI) sources, India has many 

journals in which local authors publish that are not indexed by ISI. 

(Phelan 2000, p. 593) 

In such circumstances it would be a reasonable question to ask whether the percentage 
of journals published in particular countries equates with the representation of those 
countries in the ‘international’ literature. Rather unsurprisingly, with 66 per cent of the 
journals on the ISI list published in the USA 66.8 per cent of the publications cited are 
from the USA. The UK publishes 23 per cent of the listed journals but achieves only 
some 10 per cent of publications; Russia 1 per cent of journals and 0.4 per cent of 
publications: Canada 0.6 percent of journals and 5.6 per cent of publications and 
Australia 0.6 per cent of journals but 3.5 per cent of publications. On these figures it 
would seem that the US gets about what could be expected; Russia significantly less 
than might be expected; the UK rather less than half of what might be expected and 
Canada and Australia significantly more than might be expected. On the other hand it 
might simply be that Canadian and Australian researchers are more prone to publish in 
other countries’ (especially American and British) journals. 

This data also raises the important issue of how educational debates in one country are 
related to those in others. It is not possible to tell from Phelan’s data if my surmise is 
correct, but if it is then it indicates that Canadian and Australian researchers are possibly 
more connected to educational research and policy agendas in other countries 
(especially the United Kingdom and the United States) than researchers in those 
countries are to other national debates. Extending this to the Indian example used by 
Phelan the question arises as to how the significant number of publications and journals 
that are produced in India are related to the ‘international’ literature. Or, more bluntly, 
in what ways are the journals on the ISI list actually in their content and orientation 
‘international’? Or, even more importantly, in what sense other than the rather limited 
example given above of Canadian and Australian researchers is there a truly 
‘international’ educational research literature? 


59 


RICHARD BATES 


There is also the important question of the impact of publication. Does anyone read 
the papers published? One way of attempting to address this issue is to look at the 
citation rates of articles and journals. Phelan reports an ‘impact’ score for individual 
journals which is ‘the average number of citations per publication for that journal over 
the years it has been indexed by ISF (Phelan 2000, p. 588). This would presumably 
mean that an impact score of 1.00 indicates that there would have been, on average, 
one citation per publication. For the years 1981-97 fifty-one of these journals had an 
impact score of less than one. Two had an impact score of zero! It would seem 
difficult to argue that these journals were publishing significant research in education, 
but they remain on the ISI list. 

Turning to national comparisons in citation rates as an index of the impact of 
educational research Phelan provides data (table 3.4, p. 600) that show significant 
fluctuations over time. For instance, for the period 1987-98 Denmark tops the list with 
an average citation rate of 4.27. But this is almost wholly achieved during the period 
1987-91 with an average of 10.19 citations per publication (1 st place) while for the 
period 1993-97 the rate fell to a relatively low 1.03 (11 th place). Sweden likewise had 
an overall score of 2.93 due largely to a high score (7.38, 2 nd place) in the early period 
falling to a low score (0.73, 22 nd place) in the later period. The United States rates third 
on the list for the overall period 1987-98 (2.71) but again most of this is due to a 
higher rate (4.12, 5 th place) in the early period, while for the later period the rate was 
2.17 (3 rd place). Australia is listed tenth on this table for the overall period (2.12) with 
an early score of 3.39 (8 th place) and a later score of 1.08 (7 th place). 

Phelan’s conclusion here is interesting: 

Looking at the data over time, however, it is clear that Denmark and 
Sweden were strongest on this measure in the period from 1987 to 
1991. Their best research, on average, was produced in the 1980’s and 
the quality of their research by this measure appears to have declined 
in more recent years. (Phelan 2000, p. 599) 

The leap here from ‘rate of citation’ to ‘quality of research’ does not seem justified. 
One is reminded of the huge citation rates that followed the initial publication of 
claims to have achieved ‘cold fusion’ and the eventual disconfirmation of the original 
research. 

Phelan’s comment on Australia is that ‘the ability to attain a relatively large number 
of citations per article produced appears to be Australia’s weakest area in the data 
examined here’ (Phelan 2000, p. 599). 
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Turning to internal, Australian, rather than international comparisons Phelan 
examines the performance of 38 Australian universities on a variety of measures: 
share of publications, historical share of citations, share of citations to recent 
publications, recent publication impact, publications per staff member and citations 
to recent work per staff member. Combining data for these measures Phelan 
constructs a productivity index. His conclusions are wholly based on the ISI data and 
are limited to the 25 percent of Australian educational research published in those 
journals. However, his claim is that publication in ISI journals represents both 
international recognition and an index of quality. 

His conclusions are clear: 

The data [show] that a relatively small number of universities are located 
near the top on a large range of productivity measures presented here. 

The University of Queensland and Curtin University of Technology 
perform particularly strongly on most of the measures examined here. 

Deakin University, the University of Sydney, and the University of New 
South Wales, all of which are also very strong on a number of measures 
presented here, closely follow them. Other universities that clearly 
make a major contribution to internationally recognised educational 
research are the University of Western Australia, La Trobe University, 
Queensland University of Technology, the University of Western 
Sydney, and Monash University. (Phelan 2000, p. 6l4) 

He also passes comment on the apparent rise and fall of particular universities. 

The most current of the measures presented here suggest that three 
universities, Deakin University, the Queensland University of 
Technology, and the University of Western Sydney have emerged 
particularly strongly in recent years to become leaders in the area of 
educational research. The University of Sydney still performs well on 
many measures, but its performance appears to be diminishing over 
time. The performances of the University of New England, the 
University of Adelaide, Macquarie University, and the Australian 
National University have also weakened substantially in this field over 
the years. (Phelan 2000, p. 6l4) 

However he also argues that there is a clear separation between research-producing 
institutions and the rest. 
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All of the institutions mentioned so far, however, even the declining 
ones, considerably outperform a fairly large group of universities at the 
bottom of the distribution . . . that have not produced much in the way 
of educational research over a long period of time. (Phelan 2000, p. 6l4) 

One of the most telling charts however is the one that relates research productivity 
as measured by publication rates to research expenditures. Here Phelan’s conclusion 
is that 


The University of Queensland, Queensland University of Technology, 

Curtin University, the University of Sydney and Monash University, La 
Trobe University and Deakin University . . . produce more research than 
would be predicted by their levels of expenditure on research. The 
University of Melbourne, Griffith University, Macquarie University and the 
University of Wollongong, all universities with reasonably large 
expenditures on educational research, produce fewer articles than would 
be expected from the amount of money invested. (Phelan 2000, p. 649) 

If citation rates are taken as a measure of attention paid to research these comparisons 
become even more striking. To take but two extreme examples. Melbourne University 
accounts for 10.77 per cent of the total expenditure on educational research but achieves 
4.7 per cent of publications and 0.33 per cent of citations. Deakin University, with 2.31 
per cent of educational research expenditure, achieves 4.6 per cent of publications and 
3.89 per cent of citations for the period 1992-97 (Phelan 2000, pp. 649-51). 

It is perhaps worth noting here that the ARC research quantum, which ranks 
universities by overall research performance, significantly diverges from the ranking 
provided by Phelan’s data! For instance, if universities are ranked by 2001 research 
operating grants as an index of research quantum and their ranking compared with 
Phelan’s composite index of productivity in educational research (Table 1 - overleaf) 
some fairly startling disparities become apparent. For instance of the 18 universities 
where educational research ranked higher than the university’s overall ranking 6 were 
6-10 ranks higher and 3 were 11-15 ranks higher. Of those 19 universities where 
educational research ranked lower than the overall university ranking 4 were 6-10 
ranks lower, 1 was 11 ranks lower and 3 were more than 16 ranks lower! This 
suggests what most of us have believed for some time: that the crude and cheap 
Australian way of ranking universities by overall performance provides a highly 
inaccurate estimation of the productivity of particular fields of research and that a 
system similar to that employed in England where like fields of research are 
compared with like would produce significantly different results. 
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University 

Overall rank 

Education 

Difference 

Sydney 

1 

4 

-3 

Monash 

2 

10 

-8 

Queensland 

3 

1 

+2 

ANU 

4 

23 

-19 

Melbourne 

5 

13 

-8 

QUT 

6 

8 

-2 

uws 

7 

9 

-2 

UNSW 

8 

5 

+3 

Griffith 

9 

20 

-11 

La Trobe 

10 

7 

+3 

RMIT 

11 

33 

-22 

UniSA 

12 

28 

-16 

Deakin 

13 

3 

+10 

Curtin 

14 

2 

+ 12 

UTS 

15 

16 

-1 

Newcastle 

16 

11 

+5 

Edith Cowan 

17 

22 

-5 

Adelaide 

18 

12 

+6 

VUT 

19 

25 

-6 

Charles Sturt 

20 

30 

-10 

UWA 

21 

6 

+15 

Macquarie 

22 

14 

+8 

Tasmania 

23 

21 

+2 

James Cook 

24 

17 

+7 

Wollongong 

25 

26 

-1 

Flinders 

26 

18 

+8 

UNE 

27 

19 

+8 

CQU 

28 

27 

+1 

USQ 

29 

24 

+5 

Murdoch 

30 

15 

+15 

ACU 

31 

34 

-3 

Swinburne 

32 

36 

-4 

Canberra 

33 

31 

+2 

Southern Cross 

34 

37 

-3 

Ballarat 

35 

38 

-3 

NTU 

36 

31 

+5 


Table 1 : University and education rankings compared 
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Phelan provides interesting data in a number of other areas including the 
contributions of individual authors and individual publications and rates of 
collaboration. But perhaps the most significant recommendation he makes is that an 
index such as that produced by ISI for ‘international’ publications should be devised 
for Australian educational research as a whole, including the 75 per cent that is 
currently not part of the ISI database. It would seem useful for AARE to pursue this 
possibility. 

Overall the Phelan study, despite the limitations of the ISI data noted above, provides 
a useful (and the only available) estimate of the international impact of Australian 
educational research and the comparative contributions of various institutions. Two 
particular issues emerge from the study. Firstly, the inappropriateness of the ARC 
research quantum ranking by institution rather than by field of study is clearly shown. 
Secondly, the need for an Australian-based index comparable to that of the ISI but 
inclusive of all Australian educational research would be a significant contribution to 
a proper assessment of the health and impact of Australian educational research. 


Notes 

1 These were education, mathematical sciences, physical sciences, chemical sciences, 
earth sciences, biological sciences, information science, engineering, agricultural 
sciences, medical sciences, economics, political sciences and behavioural sciences. 

2 It can be argued of course that if a country produces a large number of high 
circulation journals it will inevitably constitute a larger proportion of total world 
literature in the field. Whether or not this by itself ensures that such a contribution 
to the total world literature is indeed international in its focus, scope or interest is 
another judgement entirely. 
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